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DCI/ICS 81-2234 
11 December 1981 


MEMORANDUM FOR: rs 
irector, Intelligence Community Staff 


VIA: 
Director, Office of Assessment and Evaluation 


and Evaluation 


SUBJECT: Intelligence User Survey 


Oe acai i ea 7 STAT 


close coordination with As you are aware, it was not possible 
for us on this staff to impose a rigorous methodology for analyzing the 
data on the User Survey prior to the submission of the questionnaires to 
respondents for the following reasons: 


a. The questionnaire itself represented an amalgam of disparate 
interests of the IC Staff (PAO and PGS), DIA, NSA, and NFAC. 
Although the Office of Medical Services "experts" on such 
questionnaires attempted to ensure some form of rigor which would 
enable analysts to evaluate the meaning of the data acquired, their 
efforts were only partially successful. 


b. Although the Survey was sponsored by you as D/ICS and by 
John McMahon as D/NFAC (wearing his Community hat), the form of the 
questionnaire and the manner in which the project was implemented was 
skewed by the virtual veto power of din his role as STAT 
Chairman of the agency Community on Intelligence production. 
Understandably, mag to do nothing to antagonize members of 


his Community Working Group, which represents one of the few attempts 
of NFAC to exercise a Community role. 
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SUBJECT: Intelligence User Survey 


2. It is now time for us to prepare for you a report summarizing our 
initial findings based on a preliminary analysis of the data obtained both 
from the questionnaires and the interviews. Once this has been done, we 
would hope to subject the data to far more rigorous analysis along the 
Jines indicated in the attachment. If you approve of this method and 
approach, we would hope to have the initial report for you by the end of 
January. We would also welcome your thoughts on the model we have 
constructed as a guide to further analysis. This is provided in the 
attachment. 


Attachment 


Approved For Release 2005/03/24 : CIA-RDP83M00171R001500050002-5 


STAT 


, Approved For Release 2005/03/24 : CIA-RDP83M00171R001500050002-5 


SUBJECT: Intelligence User Survey 


Distribution: DCI/ICS 81-2234 
Orig-D/ICS 

1-DD/ICS 
1-D/OA&E 
1-A-D/OP 
1-NFAC 

STAT 1-0P ( 
1-0A&E 
1-OA&E 
1-0A&E 
1-OA&E Subjec 
1-OA&E Chrono 
1-ICS Registry 


STAT ja ns 0: /81) 


Approved For Release 2005/03/24 : CIA-RDP83M00171R001500050002-5 


Approved For Release 2005/03/24 : CIA-RDP83M00171R001500050002-5 


14 December 1981 


Proposed Approach to Evaluating Consumer 
Survey Questionnaire Data 


Background 


A survey addressing the interests and preferences, intelligence usage, 
and derived satisfactions of Washington-level policy consumers has been 
underway for several months under the sponsorship of the Interagency Working 
Group on Intelligence Production. Collection of data in questionnaire form 
is complete. Follow-on interviews of select respondents are now being 
conducted. Computerized questionnaire data, in their entirety, have been 
provided to the IC Staff to support whatever evaluation purposes may be 
deemed worthwhile. Simply put, the questionnaire asked Carter Administration 
incumbents about: 


What they wanted. 

What they got, and how they got it. 
How much of it they used. , 

How they liked it. 


Objectives of the Analysis 
To structure, assess, and understand the consumer survey so that we can: 


e Estimate the value of intelligence information to the consumer. 
e Find ways to make intelligence more useful. 


Analytical Concept 


Establishment of a comprehensive, sustained product evaluation function, 
possibly within the re-organized Intelligence Community Staff, has been under 
consideration by Community management for some time. Since any such activity 
might benefit from the existence of a logical analytical infrastructure upon 
which to proceed, this paper proposes a systematic, in-depth approach to 
interpreting the Survey data. Our approach is aimed at defining and 
describing the filtering process through which intelligence outputs progress 
toward the ultimate payoff: application by a consumer against an issue he 
faces, with results beneficial to U.S. national interests (See Attachment 
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Data extracted from the survey questionnaires will be used to estimate values 
and frequencies at different nodes in the intelligence utilization process. 
Fuller appreciation of this process, and how intelligence is flowing through 
it, ought to suggest promising areas for modifying procedures or resource 
allocations so as to increase the proportion of useful outputs. 


Since the survey is essentially a market poll, the proposed analysis 
will seek-out and present statistical highlights in a "polling report" vein. 
Our initial efforts will be of this sort, and should find a ready audience 


among the functional managers directly concerned with intelligence production 
activities. 


Analytic Sequence 
It is proposed that IC Staff analysis proceed along the following lines: 
Phase I: Concept Development and Exploratory Data Analysis 
e Examine data for general patterns, realizing that more 
sophisticated analysis will follow. Simplify descriptions without 
concern for uncertainty or variability. 


e Summarize responses to questions deemed of particular interest. 


e Develop the appropriate statistical methods and data extraction 
tools to circumvent methodological difficulties (See Attachment 2). 


Phase II: Refinement of the Model (an iterative process) 


e Determine which aspects of the model are addressed by data from the 
questionnaire, and what data is missing. 


e Examine, one-by-one the hypothesis implicit in the model. Prune 
a the model if data do not support hypotheses (See Attachment 
3 e 7 

Phase III: Quantification of the Model 


e Associate specific questions in the questionnaire with nodes in the 
model. 


e Verify the "closeness" of questions with appropriate statistical 
tools, making adjustments as necessary. 
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e For each point in the model, estimate frequencies, values, 
proportions and levels of significance as required. 


e Describe activity levels through the utilization process; 
explain routing, conduct sensitivity analysis to identify 
chokepoints and opportunities for increasing the utility of 
intelligence. 

Phase IV: Prepare Final Report 

e Dovetail statistical results with textual and verbal responses. 

e Present inferences drawn from Survey data. 

e Recommend resource reallocations and procedural changes that 
offer the best opportunity to maximize the utility of the 
intelligence product to consumers. 


e Recommend directions for further study (those points in the 
model for which data is inconclusive or non-existent). 
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Attachment 1. Prototype Model of the Utilization Process 


This attachment offers for critical review initial returns from an 
OA&E effort to develop a logical model of the process by which consumers 
utilize the outputs of the Intelligence Community. Confronted with the 
problem of interpreting data collected via the Consumer Survey 
Questionnaire, one cannot avoid the impression that here is a wealth of 
fact and opinion pertinent to the "bottom line" of the intelligence 
business, which ought to be of utmost interest to all levels of Community 
management if only the bits and pieces can be put together in some 
coherent fashion. What we have is a bunch of empirical observations in 
search of theoretical structure. The analysis which follows endeavors to 
prototype such a structure, without which the Survey "results" can 
constitute little more than titillating tidbits of statistical fact, left 
to drift aimlessly in a contextual void (poetic, huh!). 


Figure I depicts a detailed model of the intelligence utilization 
process. RESOURCES are transformed by intelligence SUPPLIERS into OUTPUT, 
which can be characterized according to FOCUS (i.e., geographical and 
topical concern), KIND (i.e., raw intelligence, in-depth analysis, etc.), 
and FORM (i.e., formal publication, especially prepared briefing, etc.). 
These outputs are in demand by certain CONSUMERS who have individual 
INTERESTS AND PREFERENCES which can also be characterized by focus, kind 
and form. A consumer demands intelligence outputs, and is willing to 
expend time and effort in their consumption, in the expectation that they 
will promote beneficial OUTCOMES to issues of importance he faces. For 
this to happen, suppliers must make the right output AVAILABLE to the 
consumer, who must then proceed to ACQUIRE, ASSIMILATE and APPLY that 
output. An output's journey from the dock of an intelligence supplier to 
beneficial application by a consumer against an important issue is clearly 
long and tortuous. Numerous opportunities exist for each output to go 
astray, or "leak-out" of the system, short of its final destination. 


Within the logic of the model, the efficiency with which the 
Community performs is represented in the proportion of output which 
completes the passage to beneficial application. Only these survivors 
contribute directly to the credit side of the Community's ledger. Al] 
other output represents only costs, which take the form of consumers' time 
and effort as well as resources put into the intelligence production 
process. Any action which increases the probability that a given output 
will move in the right direction across any one of the many decision 
"switches" depicted in the model will, if nothing else changes, serve to 
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increase performance efficiency. Since most of these decisions are made 
in the consumption sector, either by consuming principals themselves, or 
their staff, it is necessary for suppliers to discover the criteria upon 
which these decisions are based. 


Intuitition suggests a consumer would consider at least three 
sequential aspects of an intelligence output in deciding whether or not to 
proceed to the next step in the utilization process: RELEVANCE (i.e., how 
closely its focus, kind and form matches his interests and preferences); 
INTRINSIC MERIT (7.e., his perception of the competence, or credibility, 
of its content); and COMPLETENESS (i.e., how thoroughly it covers his 
interest in the matter at hand). All of this, of course, presumes that 
the consumer perceives a NEED for information or analysis about something, 
and harbors some EXPECTATIONS that intelligence will contribute toward 
that need's gratification. 


An abridged version of the utilization process is shown in Figure II, 
with the twenty-six questions asked in the Survey Questionnaire 
superimposed to indicate that phase of the process to which each question 
applies. Two essentially different kinds of questions are identified: 
those which call upon the consumer for purely descriptive estimates (e.g., 
How often did...), and those which ask for an evaluation (e.g., How 
satisfied were you with...). The plan is to check these “common sense" 
results against question associations derived independently via various 
statistical techniques, making whatever refinements to the model appear to 
be warranted. Figure II depicts the abridged model in a simple yes-no 
probability (or decision) tree format. Each decision point (denoted by 
a~m”) is further modeled as a four-part sequence, as indicated in Figure 
IV. Data collected via the Consumer Survey will be assessed in the 
context of the mathematical expressions associated with these diagrams 
(still to be worked out). 
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Attachment 2. Methodological Difficulties 


In the questionnaire phase of the Consumer Survey, over 130 
individuals representing 13 organizations and 4 levels of responsibility 
have responded to over 150 queries concerning the frequency, quantity, 
quality and usefulness of intelligence output. The raw responses exceed 
20,000. Since the volume of data is so large, we must place a great deal 
of effort on selecting statistically sound methods that will allow us to 
reduce and summarize the data while permitting us to draw valid 
inferences. This task is made more difficult because the survey was not 
designed with a clear line of analysis in mind. 


Three areas particularly hamper present efforts at interpretation: 


e Lack of a clear experimental design: A sound experimental 
esign, even with masses of data this large, goes a long way 

toward making it simple to analyze the data. Any biases or 
interactions can be identified, accounted for, summarized, and 
tested for significance. An experimental design tied to a 
specific analytical approach is essential to reduce error and to 
negate the effect of extraneous variables. The Consumer Survey 
appears to have given only cursory consideration to such 
fundamentals as the choice and range of the response variables, 
minimizing distortions from extraneous variables, and the 
make-up of the consumer population. Nonparametric* cluster 
analysis and resistant** classification analysis offer some hope 
for making reasonable assumptions and associations, but drawing 
inferences will require the utmost caution. We are not in 
control of the data any more than cattle drivers are in control 
of a stampede. It is difficult to tell what direction the 
cattle are heading, and estimates of the time of delivery and of 
the profits upon shipment are specious. 


e Missing data: Most consumers did not answer every question. In 
Fact, there are more than 4500 missing data points--about 20% of 
the data. A comparison between two questions can be considered 
fair only if the groups of consumers responding to each question 
are similar to each other and representative of 


*Nonparametric (or robust) techniques are relatively insensitive to 
data distribution. 


**Resistant measures are those which are relatively insensitive to 
large variations in small portions of the data. 
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the larger population. Similarly, the comparison of two 
consumers’ views must be adjusted to account for 
unrepresentativeness in the sample of questions. Missing data 
greatly decreases the likelihood that fair comparisons can be 
made without some form of adjustment. Tukey's algorithms for 
direct and indirect standardization, developed for use in poorly 
controlled studies, are good methods for comparing responses in 
two or more disproportionate categories. 


e Inequitable scales: Once data has been standardized and 
fegitinately extracted from the questionnaire, it may still be 
difficult to compare or summarize the data. Rating scales for 
many of the questions are inherently incompatible. One question 
partitions the frequency domain into "Daily", "Weekly", and, 
'Monthly"; another into "Frequently", "Occasionally", "Rarely"; 
a third into "Excessive", "Sufficient", "Insufficient". Robust 
and resistent measures that adjust for the effect of broad 
categories may allow for order-of-magnitude comparisons. 


Each of these problems will complicate our analysis, and the 
synergistic effect of all three could invalidate classical correction 
technqiues. The degree to which these complications will affect our 
understanding of the survey and the subsequent evaluation of the 
intelligence product is not yet clear though there is a reasonable 
prospect for extracting useful data. Furthermore, the nonparametric and 
the resistant methods outlined here assure that if we can move ahead, it 
will be in a generally correct direction if at the expense of some 
precision and sensitivity. 
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Attachment 3: Refinement of the Model 


The ICS approach to analyzing the Consumer Survey data will combine a 
clear theoretical framework with cautious, step by step, refinement of the 
framework. 


Ideally, before an experiment or survey is conducted, with two 
issues should be dealt with somewhat rigorously: 


the model - the series of interrelated hypotheses and assumptions, 
which will be proved or disproved based upon the experimental data; 
and, 


the experimental design - the structure that allows us to tie the 
data collection to the hypotheses in our model. 


Since these issues were not integrated into the consumer survey a priori, 


we must impose them externally and a posteriori. While a posteriori 
designs do not invalidate analyses they can make them more controversial. 


Our analytical approach must therefore use the model and exploratory 
statistics to support each other. The attached flow diagram outlines an 
interactive procedure we might follow. The model will set forth an 
hypothesis. Data exploration will yield indications of both the truth of 
our hypothesis and of the direction confirmation analysis ought to take. 
If confirmatory analysis supports our hypothesis we proceed to the next 
hypothesis. If however, our hypothesis should prove false, we should make. 
the appropriate adjustments to our model, then retest all hypotheses. 
Only when we have run through all hypotheses without rejection can we 
expect to fully understand the survey data and feel confident about 
inferences we draw from it. 
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Fiow Diagram. Suggested Analytical Procedure to Evaluate Consumer 
Survey Data. 
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